Skip to content

Fix test runner not working for tests using relative paths in workspace projects#2236

Merged
dan-niles merged 6 commits into
wso2:release/ballerina-5.12.xfrom
dan-niles:fix-test-runner-in-workspaces
May 19, 2026
Merged

Fix test runner not working for tests using relative paths in workspace projects#2236
dan-niles merged 6 commits into
wso2:release/ballerina-5.12.xfrom
dan-niles:fix-test-runner-in-workspaces

Conversation

@dan-niles
Copy link
Copy Markdown
Contributor

@dan-niles dan-niles commented May 13, 2026

Purpose

Related to wso2/product-integrator#1569

This PR fixes the test runner not working when using relative paths in tests. This updates the test runner to cd into the relevant packages before executing the bal test command.

Screen.Recording.2026-05-17.at.8.03.47.PM.mov

Summary by CodeRabbit

  • Bug Fixes

    • Validate and clean projects before running tests to improve reliability.
    • Test results are now captured and displayed even when test runs exit with failures.
  • Improvements

    • More reliable discovery and loading of generated test reports from test output.
    • Simplified working-directory behavior and more consistent command execution for test runs.

Review Change Stack

@coderabbitai
Copy link
Copy Markdown
Contributor

coderabbitai Bot commented May 13, 2026

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 436697de-9af4-4837-80d8-4a179291ba07

📥 Commits

Reviewing files that changed from the base of the PR and between 587c4e5 and 6f1264f.

📒 Files selected for processing (1)
  • workspaces/ballerina/ballerina-extension/src/features/test-explorer/runner.ts
🚧 Files skipped from review as they are similar to previous changes (1)
  • workspaces/ballerina/ballerina-extension/src/features/test-explorer/runner.ts

📝 Walkthrough

Walkthrough

The test runner now cleans/validates each involved project before running tests, unifies command building and working-directory handling, always resolves command execution returning an exitCode, and extracts test_results.json paths from bal test stdout for result reporting.

Changes

Test Runner Refactoring

Layer / File(s) Summary
Project path parsing and imports
workspaces/ballerina/ballerina-extension/src/features/test-explorer/runner.ts
Add cleanAndValidateProject import and update getProjectPathFromTestItem to reconstruct test: IDs into project paths that may contain :.
Pre-test cleanup and validation
workspaces/ballerina/ballerina-extension/src/features/test-explorer/runner.ts
Collect unique project paths from selected test items and call cleanAndValidateProject per project, logging failures without aborting the run.
Command builder refactor
workspaces/ballerina/ballerina-extension/src/features/test-explorer/runner.ts
buildTestCommand drops the workspace projectName parameter and always builds commands assuming a fixed CWD; supports eval vs coverage commands and optional --tests.
Workspace working-dir logic removal
workspaces/ballerina/ballerina-extension/src/features/test-explorer/runner.ts
Remove getProjectNameIfWorkspace and workspace-root vs project-root branching so commands use project paths directly.
Test execution branches refactored
workspaces/ballerina/ballerina-extension/src/features/test-explorer/runner.ts
Project-group, test-group, and single-test branches compute isEval, run bal test via runCommand, extract report paths from stdout, and call handleEvalReport or reportTestResults (with reportPathOverride / individualTest).
Reporting API extension
workspaces/ballerina/ballerina-extension/src/features/test-explorer/runner.ts
reportTestResults accepts an optional reportPathOverride to load test_results.json directly from the extracted path.
runCommand semantics and report extraction
workspaces/ballerina/ballerina-extension/src/features/test-explorer/runner.ts
runCommand now returns { stdout, stderr, exitCode } and always resolves on process close; extractTestReportPath parses bal test stdout to locate generated test_results.json from the "Generating Test Report" line.

🎯 4 (Complex) | ⏱️ ~45 minutes

Suggested reviewers:

  • hevayo
  • gigara

"I hopped through code with nimble feet,
Cleaned each project before they'd meet,
I listened to Bal's test stream and found the trace,
Now reports arrive in their proper place,
A tiny rabbit cheers this tidy race."

🚥 Pre-merge checks | ✅ 3 | ❌ 2

❌ Failed checks (2 warnings)

Check name Status Explanation Resolution
Description check ⚠️ Warning The description is largely incomplete; it only includes the Purpose section while omitting most required template sections (Goals, Approach, UI Component Development, Security checks, Test environment, etc.). Complete the PR description by adding the missing template sections such as Goals, Approach, Automation tests, Security checks, and Test environment to provide comprehensive context for reviewers.
Docstring Coverage ⚠️ Warning Docstring coverage is 37.50% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (3 passed)
Check name Status Explanation
Title check ✅ Passed The title accurately describes the main change: fixing test runner issues with relative paths in workspace projects, which aligns with the file modifications and PR objectives.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Warning

There were issues while running some tools. Please review the errors and either fix the tool's configuration or disable the tool if it's a critical failure.

🔧 ESLint

If the error stems from missing dependencies, add them to the package.json file. For unrecoverable errors (e.g., due to private dependencies), disable the tool in the CodeRabbit configuration.

ESLint skipped: no ESLint configuration detected in root package.json. To enable, add eslint to devDependencies.


Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@dan-niles dan-niles requested a review from Copilot May 13, 2026 18:58
@dan-niles
Copy link
Copy Markdown
Contributor Author

@CodeRabbit review

@coderabbitai
Copy link
Copy Markdown
Contributor

coderabbitai Bot commented May 13, 2026

✅ Actions performed

Review triggered.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (1)
workspaces/ballerina/ballerina-extension/src/features/test-explorer/runner.ts (1)

230-246: ⚡ Quick win

Run pre-test project cleanup concurrently to avoid serializing LSP round-trips.

When a run touches multiple projects (common for workspace runs — exactly the scenario this PR targets), cleanAndValidateProject is awaited one project at a time, so each LSP round-trip blocks the next. Since the per-project cleanups are independent and errors are already swallowed/logged, Promise.allSettled gets the same semantics with parallelism.

Additionally, consider checking token.isCancellationRequested before/while cleaning so users can abort during the pre-test phase (today cancellation is only honored inside the per-test forEach below).

♻️ Suggested refactor
-    const langClient = extension.ballerinaExtInstance.langClient;
-    for (const projectPath of projectPaths) {
-        try {
-            await cleanAndValidateProject(langClient, projectPath);
-        } catch (err) {
-            console.error(`Failed to clean project before test run: ${projectPath}`, err);
-        }
-    }
+    const langClient = extension.ballerinaExtInstance.langClient;
+    await Promise.allSettled(
+        Array.from(projectPaths).map(async (projectPath) => {
+            try {
+                await cleanAndValidateProject(langClient, projectPath);
+            } catch (err) {
+                console.error(`Failed to clean project before test run: ${projectPath}`, err);
+            }
+        })
+    );
+    if (token.isCancellationRequested) {
+        include.forEach((test) => run.skipped(test));
+        run.end();
+        return;
+    }
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In
`@workspaces/ballerina/ballerina-extension/src/features/test-explorer/runner.ts`
around lines 230 - 246, The current serial pre-test cleanup loops over
projectPaths and awaits cleanAndValidateProject for each, causing serialized LSP
round-trips; change this to run cleanAndValidateProject concurrently using
Promise.allSettled over an array of promises created from projectPaths (keep
using extension.ballerinaExtInstance.langClient and the existing
cleanAndValidateProject function), and before starting and/or inside each
cleanup promise check token.isCancellationRequested to short-circuit cancelled
runs; preserve the existing error-logging behavior (console.error) for failures
since allSettled will return results you can inspect and log.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In
`@workspaces/ballerina/ballerina-extension/src/features/test-explorer/runner.ts`:
- Around line 230-246: The current serial pre-test cleanup loops over
projectPaths and awaits cleanAndValidateProject for each, causing serialized LSP
round-trips; change this to run cleanAndValidateProject concurrently using
Promise.allSettled over an array of promises created from projectPaths (keep
using extension.ballerinaExtInstance.langClient and the existing
cleanAndValidateProject function), and before starting and/or inside each
cleanup promise check token.isCancellationRequested to short-circuit cancelled
runs; preserve the existing error-logging behavior (console.error) for failures
since allSettled will return results you can inspect and log.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 45164d2a-ca5e-4bf5-a3df-6b66c45fba48

📥 Commits

Reviewing files that changed from the base of the PR and between afd0331 and 643177b.

📒 Files selected for processing (1)
  • workspaces/ballerina/ballerina-extension/src/features/test-explorer/runner.ts

Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR updates the Ballerina test explorer runner to better support workspace projects where tests rely on relative paths, by changing how bal test is invoked and how the generated test report is located.

Changes:

  • Run bal test with the package directory as the working directory (instead of workspace root + project name argument).
  • Parse bal test stdout to extract the generated test_results.json path and use it when reporting results.
  • Align the test run flow with run/debug by invoking cleanAndValidateProject before executing tests.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@dan-niles dan-niles force-pushed the fix-test-runner-in-workspaces branch from 9827f19 to 587c4e5 Compare May 16, 2026 03:32
@dan-niles dan-niles marked this pull request as ready for review May 17, 2026 14:35
@dan-niles dan-niles requested review from gigara and hevayo as code owners May 17, 2026 14:35
Comment thread workspaces/ballerina/ballerina-extension/src/features/test-explorer/runner.ts Outdated
@dan-niles dan-niles merged commit 38f1b74 into wso2:release/ballerina-5.12.x May 19, 2026
8 checks passed
@dan-niles dan-niles deleted the fix-test-runner-in-workspaces branch May 19, 2026 05:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants